From Local to Global Patterns: Evaluation Issues in Rule Learning Algorithms

Separate-and-conquer or covering rule learning algorithms may be viewed as a technique for using local pattern discovery for generating a global theory. Local patterns are learned one at a time, and each pattern is evaluated in a local context, with respect to the number of positive and negative examples that it covers. Global context is provided by removing the examples that are covered by previous patterns before learning a new rule. In this paper, we discuss several research issues that arise in this context. We start with a brief discussion of covering algorithms, their problems, and review a few suggestions for resolving them. We then discuss the suitability of a well-known family of evaluation metrics, and analyze how they trade off coverage and precision of a rule. Our conclusion is that in many applications, coverage is only needed for establishing statistical significance, and precision is the metric that should be optimized for rules. The main problem with optimizing precision is its unreliability for low example sizes, which is mainly caused by overfitting. We then report some preliminary experiments that addresses this problem by meta-learning a predictor for the true accuracy of a rule based on its coverage on the training set.